00:00 hello everyone and thank you for joining 00:02 00:02 us today with the us 00:04 00:04 elections at the forefront of everyone's 00:06 00:06 mind the data science behind election 00:08 00:08 predictions is something that we should 00:09 00:09 all be thinking about 00:11 00:11 so thank you for joining today's 00:13 00:13 discussion where we will be focusing on 00:15 00:15 how data science can be used 00:16 00:16 to predict elections this event is part 00:19 00:19 of the think series co-hosted by the 00:20 00:20 advocate institute and echelon magazine 00:23 00:23 this will also be live streamed on the 00:25 00:25 pages of our streaming partners 00:26 00:26 echelon magazine economy next blue chip 00:29 00:29 training and consulting 00:31 00:31 news hub lk news hub lk english business 00:34 00:34 news 00:34 00:34 citizen dot lk economy and business sri 00:37 00:37 lanka 00:38 00:38 daily ft the morning the sunday business 00:41 00:41 sunday morning business the night 00:43 00:43 watchman society 00:44 00:44 court 2.0 and punchity before i 00:47 00:47 introduce our speaker for today i'd like 00:50 00:50 like to run through some very quick 00:51 00:51 housekeeping 00:52 00:52 during the lecture we'll be keeping mics 00:54 00:54 and cameras switched off to avoid 00:56 00:56 distractions 00:57 00:57 if you have any questions or points of 00:59 00:59 clarification during the session 01:01 01:01 please feel free to send them in through 01:02 01:02 the q a 01:04 01:04 option you will find an icon at the 01:06 01:06 bottom of your screen and you can send 01:07 01:07 your 01:08 01:08 questions in through that after the 01:10 01:10 lecture we will open the floor for 01:11 01:11 questions 01:12 01:12 um i would now like to introduce today's 01:15 01:15 speaker mr nuan cenarapna 01:18 01:18 he is a software engineer and computer 01:20 01:20 scientist 01:21 01:21 specializing in artificial intelligence 01:24 01:24 he is the founder of 01:25 01:25 colombolabs.inc and was the former 01:27 01:27 engineering manager at facebook from 01:29 01:29 2009 to 2017 01:31 01:31 and has an msc in computer science and 01:33 01:33 ai from the stanford university school 01:35 01:35 of engineering 01:36 01:36 uh nuan it is a pleasure to have you at 01:38 01:38 today's event 01:39 01:39 and without further ado i'd like to hand 01:41 01:41 the flow over to you 01:43 01:43 uh thanks anita for that very kind 01:46 01:46 introduction 01:47 01:47 um thanks advocate and echelon for 01:50 01:50 this opportunity uh to participate in 01:52 01:52 this discussion 01:53 01:53 and and last but not least thanks 01:55 01:55 everyone joining over zoom and facebook 01:58 01:58 live 01:58 01:58 and possibly elsewhere okay so 02:01 02:01 uh this talk is about both about 02:04 02:04 predicting elections and 02:05 02:05 about data science um so the plan is 02:08 02:08 that 02:09 02:09 we'll discuss three sort of mini case 02:11 02:11 studies 02:12 02:12 related to elections and i'll use those 02:14 02:14 as an opportunity to 02:16 02:16 also introduce some kind of basic data 02:18 02:18 science 02:19 02:19 concepts um the first case study 02:22 02:22 is about fact-checking election 02:24 02:24 predictions now now we can't really do 02:26 02:26 election predictions if we 02:28 02:28 can't tell the difference between a good 02:29 02:29 prediction and a bad prediction 02:31 02:31 so uh we will see how we can fact check 02:34 02:34 uh predictions the second case will be 02:38 02:38 about 02:38 02:38 using survey data to predict 02:42 02:42 elections and the last case will be 02:44 02:44 about using 02:45 02:45 real-time data that is data of 02:49 02:49 currently released election results to 02:52 02:52 try and predict 02:53 02:53 the end result okay let's get started 02:56 02:56 with the first case 02:57 02:57 let's assume that you have a friend who 03:00 03:00 claims to have this superpower of being 03:02 03:02 able to predict 03:03 03:03 elections let's say he says that okay i 03:06 03:06 know 03:06 03:06 the probina of the next presidential 03:08 03:08 election and let's say you're skeptical 03:11 03:11 and you ask can you prove it um and then 03:13 03:13 your friend says 03:14 03:14 i have predicted the winner of two past 03:17 03:17 elections 03:18 03:18 okay so what do you do with this 03:20 03:20 information do you 03:21 03:21 believe your friend do you not believe 03:23 03:23 your friend okay what do you do 03:25 03:25 this is where data science comes in so 03:27 03:27 so what 03:28 03:28 exactly is data science so data science 03:30 03:30 can be defined 03:31 03:31 as using scientific methods uh 03:35 03:35 on computer data uh to 03:38 03:38 uh uh to get useful information 03:41 03:41 and and and some usually increasingly 03:44 03:44 we refer to large amounts of data 03:48 03:48 the key words for me is data scientific 03:51 03:51 methods and useful information 03:53 03:53 basically you apply scientific methods 03:56 03:56 to data to get useful information 03:58 03:58 so for example you have a water survey 04:02 04:02 say and you can apply survey statistics 04:04 04:04 to get useful information 04:06 04:06 like what is the proportion of what a 04:08 04:08 certain party 04:09 04:09 is going to get that's going to be a 04:12 04:12 second case 04:14 04:14 our third case also falls into the same 04:16 04:16 pattern we are going to use current 04:18 04:18 results 04:18 04:18 of elections released so far in order to 04:22 04:22 train a real time model and using that 04:24 04:24 model we 04:25 04:25 try to predict useful information like 04:27 04:27 what's the final uh result of the 04:29 04:29 election 04:30 04:30 and we can use this same pattern to 04:32 04:32 verify your friend's claim 04:34 04:34 um we can try and see whether we can 04:36 04:36 trust him uh with predicting 04:38 04:38 the next election okay so how do we do 04:40 04:40 this 04:42 04:42 let's say we are skeptical what if your 04:45 04:45 friend is actually not a genius 04:47 04:47 but a monkey who randomly calls 04:50 04:50 elections completely randomly 04:53 04:53 and so if that monkey is two elections 04:56 04:56 there is a one in four chance 04:58 04:58 that the monkey just got lucky and came 05:00 05:00 up with the two 05:01 05:01 uh right answers okay so 05:04 05:04 statisticians call this uh monkey 05:07 05:07 hypothesis 05:08 05:08 and null hypothesis and they also call 05:11 05:11 the hypothesis that we are trying to 05:12 05:12 prove in this case hey 05:14 05:14 this your friend is a genius the 05:16 05:16 alternative hypothesis 05:17 05:17 and to choose between these two 05:19 05:19 hypotheses we do some kind of 05:20 05:20 experiment or a test and so 05:24 05:24 statisticians 05:25 05:25 basically call it the test and you pass 05:27 05:27 the test 05:28 05:28 you accept the alternative hypothesis 05:30 05:30 you fail the test 05:32 05:32 you accept the null hypothesis so 05:35 05:35 uh related to these terms is this term 05:38 05:38 statistical confidence 05:40 05:40 so statistical confidence is probably 05:42 05:42 the most important concept in statistics 05:44 05:44 uh it is the probability of failing the 05:47 05:47 test given that the null hypothesis 05:49 05:49 is actually true okay so what this 05:51 05:51 actually means is in the case of our 05:53 05:53 example 05:54 05:54 it's the probability um that uh 05:57 05:57 this prediction test is failed 06:00 06:00 uh given that your friend is actually 06:03 06:03 among who picks results at randoms 06:05 06:05 so in this case there are three 06:06 06:06 scenarios in which 06:08 06:08 uh the test will be failed uh and so 06:12 06:12 the the the statistical confidence of 06:15 06:15 this test 06:15 06:15 is three uh above four uh which is 75 06:19 06:19 percent a very similar term to 06:21 06:21 statistical confidence is statistical 06:23 06:23 significance which is sort of the 06:25 06:25 complement of confidence 06:26 06:26 it is the probability of passing the 06:28 06:28 test given that the null hypothesis is 06:30 06:30 true 06:30 06:30 so in this case it is um the case where 06:33 06:33 the uh the the the monkey by chance 06:37 06:37 happens to get the two 06:38 06:38 answers correct um and so if so if you 06:42 06:42 notice 06:43 06:43 uh significance is one minus confidence 06:46 06:46 or hundred percent 06:47 06:47 minus confidence so so so as you can 06:50 06:50 sort of intuitively understand 06:52 06:52 the most important thing that you need 06:53 06:53 to know about confidence and 06:55 06:55 significance is that high confidence is 06:57 06:57 good 06:57 06:57 but high significance is bad okay so in 07:01 07:01 our test uh 07:03 07:03 uh where your friend claimed to be a 07:06 07:06 genius by predicting two elections 07:08 07:08 uh the significance we saw was 25 07:10 07:10 percent and the confidence is 35 percent 07:13 07:13 now this 25 and 75 percent is it too 07:16 07:16 high is it too low how do we choose 07:18 07:18 so statisticians usually uh 07:21 07:21 tend to pick these limits somewhat 07:23 07:23 arbitrarily so for example significance 07:26 07:26 could be 07:27 07:27 one percent could be considered limit 20 07:29 07:29 could be considered limit it depends on 07:31 07:31 the case 07:32 07:32 but 25 on the whole seems to be 07:36 07:36 too high so so basically a test with 07:39 07:39 just these two 07:40 07:40 uh predictions is not sufficiently 07:43 07:43 significant so what we need to do 07:45 07:45 there is we need to increase the number 07:47 07:47 of tests that 07:48 07:48 the number of elections that your friend 07:50 07:50 tries to predict so for example 07:52 07:52 if your friend can correctly predict 07:54 07:54 four elections as opposed to just three 07:57 07:57 then the significance drops to 1 upon 16 08:01 08:01 which is about 1.2 percent 08:03 08:03 and confidence is now over 90 percent so 08:05 08:05 if you pick 08:06 08:06 an arbitrary line of a 10 significance 08:09 08:09 or 90 08:09 08:09 confidence then a 4 election test would 08:13 08:13 work 08:13 08:13 if you are even more strict and you want 08:16 08:16 one percent significant 08:17 08:17 which is which is 99 confidence then you 08:20 08:20 have to go with about 08:21 08:21 seven uh predictions okay so in our 08:25 08:25 current example 08:26 08:26 um so what is our conclusion um is is 08:29 08:29 your friend 08:30 08:30 a random predicting monkey or is your 08:32 08:32 friend a genius 08:33 08:33 um the correct answer is that is 08:36 08:36 actually 08:36 08:36 neither in this case we what we do is we 08:40 08:40 actually reject the test so our test has 08:43 08:43 a 08:43 08:43 significance of only 25 percent or a 08:46 08:46 confidence of 75 percent the 08:48 08:48 the significance is too high or 08:51 08:51 equivalently the confidence is too low 08:53 08:53 so 08:53 08:53 so what we should do in this case is to 08:55 08:55 actually define a different test 08:57 08:57 uh if your friend says like hey um i 08:59 08:59 predict the two elections therefore i 09:01 09:01 can print the next 09:02 09:02 you need to go back and say hey no you 09:03 09:03 need to predict more than two maybe four 09:05 09:05 or even more than that 09:07 09:07 so the most basic fact checking question 09:10 09:10 you can ask 09:12 09:12 with the election prediction but almost 09:14 09:14 any other 09:15 09:15 uh statistic is like what is the 09:18 09:18 statistical confidence of that 09:19 09:19 statistical claim 09:21 09:21 that you're making um if uh there is no 09:24 09:24 good answer to that question then 09:26 09:26 it it probably is a bit of a dubious 09:29 09:29 um conclusion okay so that's our first 09:33 09:33 case 09:33 09:33 um okay so let's move to our second case 09:37 09:37 um the second case is about predicting 09:39 09:39 elections 09:40 09:40 using surveys so um at the moment um 09:43 09:43 there's a there's a lot of excitement 09:45 09:45 about the outcome of the u.s 09:46 09:46 elections uh people who are following 09:49 09:49 uh the sort of the evolution of the 09:52 09:52 campaign before 09:53 09:53 uh election day might have seen kind of 09:55 09:55 many different predictions by different 09:57 09:57 organizations 09:58 09:58 uh this is a particularly kind of a 10:01 10:01 visually interesting 10:02 10:02 uh prediction by 538 uh so how how do 10:06 10:06 people like 538 make these predictions 10:09 10:09 so 10:10 10:10 all these predictions are based on on 10:12 10:12 polls 10:13 10:13 um various organizations do surveys to 10:16 10:16 see okay how 10:16 10:16 people might vote so this is the example 10:19 10:19 of this is actually one of the last 10:20 10:20 polls carried out 10:22 10:22 before the the the u.s presidential 10:24 10:24 election by a 10:25 10:25 by organization called morning consult 10:28 10:28 we'll come back to this later 10:30 10:30 um but first let's do a thought 10:33 10:33 experiment 10:34 10:34 to understand how polling and 10:36 10:36 particularly sampling 10:38 10:38 work so here's an example 10:41 10:41 assume there are four groups of people 10:44 10:44 each group has three waters 10:48 10:48 and the first group has three sjb 10:51 10:51 waters the second group has two sjw 10:53 10:53 voters and one slpp water the third 10:55 10:55 group has 10:56 10:56 two slpp and one sjp and the third group 10:59 10:59 has 10:60 10:60 all uh three slpp 11:03 11:03 supporters okay so you have these four 11:05 11:05 groups all have 11:06 11:06 sort of a different composition now 11:08 11:08 imagine you don't know which group is 11:10 11:10 which 11:11 11:11 but you randomly pick one of these 11:13 11:13 groups so you now randomly picked a 11:14 11:14 group it has three people 11:16 11:16 but you don't know who those three 11:17 11:17 people are 11:20 11:20 next what you do is you randomly pick 11:22 11:22 out of those three people who you don't 11:24 11:24 know 11:25 11:25 who they are you pick two people 11:28 11:28 randomly 11:29 11:29 and then you find that those two people 11:31 11:31 are one is a sjp supporter the other one 11:34 11:34 uh is an slpp supporter okay so why did 11:38 11:38 i pick this example 11:39 11:39 if you think about it this is exactly 11:42 11:42 how 11:42 11:42 sampling happens so so so the actual 11:46 11:46 population out of which you take a 11:47 11:47 sample 11:48 11:48 let's say you're trying to take a sample 11:50 11:50 from the columbus district water 11:52 11:52 population 11:53 11:53 you don't really know exactly what that 11:55 11:55 actual population looks like and how 11:56 11:56 they are going to work 11:57 11:57 so the actual population is sort of a 11:59 11:59 black box but 12:01 12:01 you know that in theory that actual 12:03 12:03 population could take all sorts of 12:05 12:05 different 12:06 12:06 uh different different compositions um 12:10 12:10 so when you take a sample out of it 12:13 12:13 uh you get some visibility into what the 12:15 12:15 actual 12:16 12:16 population looks like so for example 12:19 12:19 since we since in our sample there's one 12:21 12:21 sjp support and one 12:23 12:23 pp supporter we know that the first 12:27 12:27 uh group where all the people are sjp 12:29 12:29 supporters and the last group are 12:31 12:31 impossible 12:32 12:32 because the first one has no srpp 12:33 12:33 supporters well the last one has no 12:35 12:35 sjp supporters so we know it has to be 12:38 12:38 one of the two 12:39 12:39 middle groups and through symmetrically 12:41 12:41 we might assume that okay there is equal 12:43 12:43 chance that 12:44 12:44 it could be either of those groups so so 12:46 12:46 something 12:47 12:47 uh somewhat obvious about samples is it 12:49 12:49 tells us something useful about the 12:51 12:51 population 12:53 12:53 but the other thing about samples is 12:55 12:55 that samples are not 12:57 12:57 exactly like the population so for 12:59 12:59 example in this sample 13:01 13:01 uh the proportion of srpp voters 13:04 13:04 is one out of two which is 50 but none 13:07 13:07 of the actual populations have this 13:09 13:09 ratio 13:10 13:10 so we have one population with 33 13:12 13:12 percent as a bp and another population 13:14 13:14 with 67 13:15 13:15 but no population has 50 percent 13:18 13:18 so the point here is uh the 13:22 13:22 characteristics of the sample 13:24 13:24 need not match the characteristics of 13:27 13:27 the population exactly 13:29 13:29 and statisticians call this sampling 13:31 13:31 error so when you take a sample it won't 13:33 13:33 be exactly like the population 13:35 13:35 and so we need to keep this in mind when 13:37 13:37 we 13:38 13:38 make various conclusions about the 13:40 13:40 population using the sample as an 13:42 13:42 estimate so for example 13:44 13:44 the sample mean is not equal to the 13:46 13:46 population mean 13:48 13:48 so so another similar place is like in 13:52 13:52 our particular example 13:53 13:53 if we want to estimate the srpp water 13:56 13:56 percentage based on our sample and we 13:59 13:59 can't say 14:00 14:00 that the srpp water percentage is 50 or 14:03 14:03 because our sample has 50 14:05 14:05 using the sample we get a sense of the 14:07 14:07 distribution 14:08 14:08 of what the actual actual population 14:11 14:11 looks like 14:11 14:11 and in this case we can only say that 14:13 14:13 it's either 33 or 67 14:15 14:15 we don't say that it's 50. so we can 14:18 14:18 also represent this in a 14:20 14:20 chart so in this chart on the x-axis you 14:23 14:23 see the various possible 14:25 14:25 slpp water populations and you also see 14:28 14:28 the populations below 14:30 14:30 and on the y-axis you see the 14:31 14:31 probability of these populations 14:34 14:34 being the actual population um given 14:36 14:36 that we have observed a sample of a 14:38 14:38 certain type 14:39 14:39 so as you can see the the first and the 14:41 14:41 last are impossible and there's a 14:43 14:43 equal 50 50 chance of the two 14:45 14:45 populations in the middle 14:47 14:47 so so what i picked here is a toy toy 14:50 14:50 example with 14:51 14:51 where our population has only three 14:54 14:54 people and our sample has two people 14:56 14:56 but these com these uh conclusions 14:59 14:59 apply to a real population so 15:02 15:02 so so for example if we take a real 15:04 15:04 population like 15:06 15:06 the population of colombo uh as our 15:08 15:08 population and we take a sample out of 15:10 15:10 it 15:11 15:11 and let's say we take a sample of 100 15:13 15:13 and it turned out that our sample had 15:15 15:15 40 hjp supporters and 60 srpp 15:19 15:19 uh supporters the the the 15:22 15:22 srpp what percentage in the population 15:25 15:25 we can't conclude that it's 60 percent 15:28 15:28 we can only conclude that they'll follow 15:30 15:30 some sort of a distribution it'll 15:31 15:31 it'll it'll be around 60 percent and it 15:34 15:34 usually takes a sort of a bell shape 15:36 15:36 like this around 60 15:38 15:38 but it's not exactly 60 15:41 15:41 next suppose we do this let's draw two 15:44 15:44 lines 15:45 15:45 either side of that sort of bell shape 15:48 15:48 such that the area 15:50 15:50 in the middle bit under the curve is 15:53 15:53 about 15:53 15:53 90 of the total area under the curve 15:57 15:57 right or alternatively the the the area 15:60 15:60 under the curve 16:01 16:01 uh to the left of the left line and to 16:04 16:04 the 16:05 16:05 the right of the right line in in total 16:08 16:08 is 10 16:09 16:09 so if we if we if we draw those lines so 16:12 16:12 that the middle bit is 90 16:14 16:14 and let's assume that the lines cross at 16:17 16:17 52 16:18 16:18 and 68 percent what can we conclude 16:21 16:21 we can conclude that there is a 90 16:23 16:23 percent probability that the slpp 16:25 16:25 uh uh what percentage in the population 16:30 16:30 is between 52 and 68 right or 16:33 16:33 alternatively we can say 16:34 16:34 it's 60 plus or minus 8 16:37 16:37 so so this is this is typically how we 16:40 16:40 return an observation in a sample 16:42 16:42 into a conclusion about the population 16:46 16:46 now you might say that this this this 16:48 16:48 range is too big 16:49 16:49 52 to 28 to 68 is is 16:53 16:53 is 16 percentage points uh that's a 16:56 16:56 that's a very vague kind of estimate can 16:58 16:58 we get better 16:59 16:59 the answer is we can and what we do is 17:01 17:01 we just get a bigger sample 17:03 17:03 so if we get a sample of say a thousand 17:06 17:06 uh and we might end up with a result 17:08 17:08 like this say there are 450 sjp 17:10 17:10 supporters and 550 17:12 17:12 srpp supporters and note that in the 17:14 17:14 same population if you take two samples 17:17 17:17 you might not get the same mean 17:18 17:18 there's no reason why you should get the 17:20 17:20 same mean but anyway uh 17:22 17:22 as before we get a sort of a bell curve 17:24 17:24 but the bell is a lot sharper it's 17:25 17:25 almost like a point now 17:27 17:27 and the the error bonds are much 17:29 17:29 narrower so earlier we got 17:31 17:31 what was it um 52 to 68 17:35 17:35 but now we got 52.5 to 57.5 17:38 17:38 and plus or minus 2.5 instead of plus or 17:41 17:41 minus 17:42 17:42 8 i think it was so so 17:46 17:46 this is a general characteristic of any 17:49 17:49 estimate that you get 17:50 17:50 uh from a sample uh but you what you 17:53 17:53 would report 17:54 17:54 is actually a range and a probability 17:56 17:56 associated with 17:57 17:57 it not a a fixed number you would never 18:00 18:00 say that the srpp uh uh 18:03 18:03 water percentage in the population is 55 18:05 18:05 percent we will say it's a range 18:07 18:07 and so so these as we saw earlier these 18:10 18:10 are known as error bonds 18:12 18:12 so this is the second fact-checking 18:14 18:14 question that you could ask 18:17 18:17 what are the error bounds if someone 18:19 18:19 reports a 18:20 18:20 prediction saying hey i think there's a 18:22 18:22 people are going to get 18:23 18:23 this percentage of votes but they don't 18:26 18:26 they don't report error bonds uh it's 18:28 18:28 probably not based on statistics it 18:30 18:30 might be a guess or something like that 18:32 18:32 okay uh and uh 18:36 18:36 and all error bonds are associated with 18:38 18:38 some confidence 18:39 18:39 you could ask 90 but it doesn't have to 18:41 18:41 be 90 equal 18:43 18:43 90 95 um 18:46 18:46 as long as the confidence is kind of 18:48 18:48 reasonably high 18:50 18:50 okay so let's look at one of those early 18:52 18:52 examples that 18:53 18:53 we saw um this is a poll by 18:56 18:56 morning consult just before the 18:59 18:59 the u.s presidential election yesterday 19:02 19:02 and in the first row you see the 19:06 19:06 [Music] 19:08 19:08 a prediction about the state of arizona 19:10 19:10 and 19:11 19:11 you will see this moe three percent 19:14 19:14 uh moe stands for margin of error 19:17 19:17 and so what this means is this according 19:20 19:20 to this poll uh 19:21 19:21 joe biden leads leads 19:25 19:25 donald trump by two percentage points 19:27 19:27 but the margin of error is three percent 19:30 19:30 so what that actually means is it's 19:31 19:31 actually two plus or minus three 19:33 19:33 so it's actually two minus three to two 19:36 19:36 plus three so it's minus one which is 19:38 19:38 actually byron is behind trump 19:40 19:40 two plus five which is biden is ahead of 19:43 19:43 trump by five points 19:45 19:45 um so as as as as i mentioned earlier 19:48 19:48 this moe 19:49 19:49 is always associated with a with a 19:51 19:51 percentage confidence 19:52 19:52 in in this case it's 90 and you'll also 19:55 19:55 notice 19:56 19:56 um they've reported the size of the 19:58 19:58 sample for each of the poles 20:01 20:01 so if you take the state of florida uh 20:03 20:03 the poll has almost 20:04 20:04 4 500 people uh and you can see the 20:08 20:08 the the the states with bigger poles the 20:11 20:11 moe is smaller so 20:12 20:12 in florida the m1 is only two percent 20:14 20:14 while in colorado which has a much 20:16 20:16 smaller pole 20:17 20:17 uh the mye is uh four percent so here's 20:20 20:20 another interesting example 20:22 20:22 uh this is by the economist and here 20:24 20:24 they have they're estimating the total 20:26 20:26 electoral college what they 20:28 20:28 expect each candidate to get and they 20:30 20:30 have kind of plotted it 20:32 20:32 across time and you see that 20:35 20:35 before before around around when kovid 20:38 20:38 was kind of picking up 20:39 20:39 it was very close and since then uh the 20:42 20:42 prediction has got wider 20:43 20:43 but one thing to keep in mind here is 20:45 20:45 that the arab bars are actually quite 20:47 20:47 wide so while it might seem that biden 20:50 20:50 is a head 20:51 20:51 what you see as the light blue shading 20:54 20:54 is 20:55 20:55 the other 95 percent confidence bounds 20:58 20:58 for 20:59 20:59 for forbidden and they actually 21:01 21:01 overlapped with 21:02 21:02 with trump's 95 percent confidence bonds 21:05 21:05 which means that 21:06 21:06 there is a chance that that biden will 21:08 21:08 actually get 21:09 21:09 more electoral college um votes than 21:12 21:12 than by according to this prediction 21:15 21:15 let's look at another prediction 21:17 21:17 so this was a prediction made uh related 21:20 21:20 to the 21:21 21:21 recently concluded parliamentary 21:22 21:22 election in in 21:24 21:24 in in sri lanka and it was trying to 21:27 21:27 predict the number of seats 21:28 21:28 different parties would get in the 21:29 21:29 column electoral district 21:31 21:31 um the sample size of 1000 and and the 21:34 21:34 results were reported at a 90 confidence 21:37 21:37 level 21:38 21:38 um so everything sort of looks good on 21:40 21:40 the surface we have we have 21:42 21:42 error bounds and so on uh but but if you 21:45 21:45 remember the final results of the 21:47 21:47 election 21:47 21:47 um we see that this this poll has got 21:50 21:50 the 21:50 21:50 the the top two uh party results kind of 21:54 21:54 almost mixed up so so one uh theory that 21:57 21:57 might explain this is they actually got 21:59 21:59 the two parties mixed up 22:01 22:01 um but that's not what happened what 22:03 22:03 happened was something 22:04 22:04 almost as embarrassing so while this 22:07 22:07 poll 22:08 22:08 was supposed to be for the whole colombo 22:10 22:10 district 22:11 22:11 um the the sample data that they 22:13 22:13 collected was only from collected only 22:16 22:16 from the colombo municipal area 22:18 22:18 and so that's the the top uh 22:21 22:21 left area that you can see which which 22:23 22:23 which in terms of voting patterns is 22:25 22:25 very different from the columbus 22:27 22:27 district as a whole 22:28 22:28 so so what actually happened was when 22:29 22:29 they should have taken a random sample 22:31 22:31 the sample wasn't really random 22:34 22:34 so this is a third way in which you can 22:36 22:36 check 22:37 22:37 election predictions uh you could ask 22:40 22:40 the the analyst 22:41 22:41 whether whether they actually checked 22:43 22:43 whether the sample was random 22:45 22:45 um and it's not enough to think that 22:48 22:48 your sample is random you need to do 22:49 22:49 some sanji checks to prove that your 22:51 22:51 sample is random 22:53 22:53 and so so what are some examples of 22:54 22:54 sanity checks so in a 22:56 22:56 survey where you might ask the people 22:58 22:58 okay who are you going to vote for in 22:60 22:60 addition to asking that question you 23:01 23:01 should ask some additional questions 23:03 23:03 like 23:03 23:03 what's your age gender ethnicity things 23:06 23:06 like that some basic demographic 23:07 23:07 information 23:08 23:08 so that you can compare into the 23:10 23:10 demographics of the population that you 23:12 23:12 are trying to measure 23:13 23:13 as a whole so for example if you have a 23:16 23:16 sample and 90 23:17 23:17 of that sample are males then that can't 23:20 23:20 be right because 90 23:21 23:21 of the district are not males similarly 23:24 23:24 if the sample 23:25 23:25 has uh eighty percent of the sample are 23:28 23:28 people aged 23:29 23:29 say twenty to forty that that doesn't 23:32 23:32 doesn't sound 23:33 23:33 right either because it doesn't match 23:34 23:34 the population 23:36 23:36 another uh scientific which would have 23:39 23:39 caught this particular problem was was 23:41 23:41 ethnicity if you take the colombo mean 23:43 23:43 super council 23:44 23:44 uh in its in terms of ethnicity it's 23:46 23:46 quite quite diverse 23:48 23:48 but if you take the columbia district as 23:49 23:49 a whole that diversity kind of becomes a 23:51 23:51 lot more homogeneous 23:53 23:53 and so definitely if the this question 23:55 23:55 was asked we could have 23:57 23:57 checked the difference between the karma 23:59 23:59 municipal council and the colombo 24:01 24:01 electoral district 24:03 24:03 uh another common practice to is to get 24:06 24:06 data from 24:06 24:06 kind of various sources and so one one 24:09 24:09 sort of interesting source for data are 24:11 24:11 various social media platforms like 24:13 24:13 facebook twitter and so on 24:15 24:15 um these are very powerful data sources 24:18 24:18 because you can get data very 24:20 24:20 efficiently you can collect the data 24:21 24:21 very easily 24:22 24:22 as opposed to say phoning someone over a 24:25 24:25 poll or like 24:26 24:26 even doing polls in person so they are 24:28 24:28 very efficient at collecting data 24:31 24:31 but one problem is that the people who 24:33 24:33 tend to use this platform 24:35 24:35 tend to be again not representative of 24:37 24:37 the population so if you take 24:38 24:38 a platform like facebook it tends to 24:40 24:40 skew towards a kind of a younger 24:43 24:43 demographic and 24:44 24:44 in countries like like sri lanka and 24:47 24:47 india it tends to also skew 24:49 24:49 mail so in situations like this there 24:52 24:52 are various 24:53 24:53 statistical techniques that you can use 24:55 24:55 to correct 24:57 24:57 for this problem but uh but but in 24:60 25:00 general you need to be kind of very 25:01 25:01 careful about it it can intricately 25:03 25:03 introduce all kinds of unexpected errors 25:06 25:06 okay so so the probably the most 25:10 25:10 important the most famous example is 25:13 25:13 uh uh of of posters getting it wrong was 25:16 25:16 probably the 25:17 25:17 the the previous uh us election where 25:20 25:20 many of the posters 25:21 25:21 said that uh mrs clinton was going to 25:23 25:23 win uh 25:25 25:25 and instead mr trump won um and and 25:27 25:27 after 25:28 25:28 after the election there was a lot of 25:30 25:30 soul-searching done around okay what 25:32 25:32 what what went wrong here so i want to 25:34 25:34 raise two points one is that 25:36 25:36 one of the uh one of the reasons 25:40 25:40 uh uh for for the for the difference 25:42 25:42 between the polls and the actual results 25:43 25:43 was 25:44 25:44 uh many trump supporters were not uh 25:48 25:48 not comfortable with sharing that there 25:49 25:49 was there were 25:51 25:51 they were trump supporters so you had 25:52 25:52 these silent trump supporters who were 25:54 25:54 not getting recorded in the polls 25:56 25:56 but but to be fair by the posters if you 25:59 25:59 take this 25:60 25:60 uh poll what is actually saying is there 26:02 26:02 is a 28.6 26:04 26:04 chance that trump would win so that's 26:06 26:06 actually higher than 26:08 26:08 uh than one in four it's higher than 26:10 26:10 tossing a coin twice 26:11 26:11 and getting two heads uh so so i think 26:14 26:14 psychologically many uh many people sort 26:17 26:17 of subconsciously 26:19 26:19 interpreted this result as you know mrs 26:21 26:21 clinton is going to get 71.4 percent of 26:23 26:23 the vote which is very different from 26:25 26:25 a 71.4 percent chance of winning the 26:27 26:27 election so so i think to a certain 26:29 26:29 extent people also misinterpreted 26:31 26:31 um what what this actually meant so 26:35 26:35 so so so one last thing about these 26:38 26:38 polls 26:38 26:38 and statistics is that you could 26:41 26:41 you could you could ask a person who 26:44 26:44 they're going to vote for 26:45 26:45 they could actually honestly reply who 26:47 26:47 exactly who they think they're going to 26:48 26:48 vote for but then on election day they 26:50 26:50 could go and they could change their 26:52 26:52 mind 26:52 26:52 so so statistics makes this assumption 26:54 26:54 that the future is like the past 26:57 26:57 and all its conclusions are based on 26:60 26:60 this assumption so it's important to 27:02 27:02 note 27:02 27:02 um that this might not be the case 27:05 27:05 so that's our second case uh and let's 27:08 27:08 take a pause here and see whether we've 27:11 27:11 sorted out our q and a issues 27:13 27:13 yes thank you nuan for those two very 27:15 27:15 interesting case studies 27:17 27:17 uh q a twos have been sorted out and 27:19 27:19 we've got a bunch of questions in as 27:21 27:21 well 27:22 27:22 um so if i could just kind of 27:25 27:25 collate a couple of questions because 27:27 27:27 some asking fairly similar questions 27:30 27:30 one question that has come through is 27:33 27:33 how is data 27:34 27:34 collected and how accurate is it um 27:37 27:37 and don't you need very large amounts of 27:38 27:38 data to do data science properly 27:42 27:42 would you like to tackle that one yeah 27:44 27:44 so 27:45 27:45 um let me answer this this question 27:48 27:48 specifically 27:49 27:49 with respect to elections and and then 27:51 27:51 we might actually generalize it 27:53 27:53 so on the on the question 27:56 27:56 of how much data i think this slide 27:59 27:59 which has the morning consult poll gives 28:01 28:01 you some indication 28:03 28:03 so how much data you need to collect 28:06 28:06 depends 28:07 28:07 on the the the how accurate you want the 28:10 28:10 poll to be 28:11 28:11 so for example if you take a state like 28:14 28:14 colorado 28:16 28:16 one of the reasons why the poll was 28:18 28:18 quite small 28:19 28:19 is that biden was her head by a large 28:21 28:21 amount so that 28:22 28:22 you're you're fine if the margin of 28:24 28:24 error is kind of quite high so that you 28:26 28:26 can afford 28:27 28:27 a kind of a lower sample size on the 28:29 28:29 other on the other hand if you take a 28:31 28:31 state like florida where 28:32 28:32 the the election is much closer you want 28:34 28:34 to get a lot more precise 28:36 28:36 so so the answer also depends on how 28:38 28:38 precise you want to be 28:40 28:40 if you uh if if the current data is 28:43 28:43 it doesn't give you a precise enough 28:44 28:44 answer then you might need to increase 28:46 28:46 it but the opposite might also be true 28:48 28:48 you might have 28:48 28:48 too much data than you need to make to 28:51 28:51 answer the question 28:52 28:52 that you want to answer so it depends on 28:55 28:55 the the question you want to answer so 28:58 28:58 in general there is this sort of 28:60 28:60 um there is this sort of a kind of ex 29:02 29:02 like 29:03 29:03 diminishing returns that you see with 29:05 29:05 kind of larger and larger samples 29:06 29:06 so what i mean by that is let's say 29:10 29:10 you have a sample of a thousand and you 29:12 29:12 have a sample of two thousand 29:14 29:14 um and then you say you have a sample of 29:17 29:17 of of three thousand uh so the 29:19 29:19 difference between the thousand and the 29:21 29:21 two thousand is probably more than the 29:22 29:22 difference between the two thousand and 29:24 29:24 three thousand or even the three two 29:25 29:25 thousand and the four thousand 29:26 29:26 so so so usually that there is some 29:29 29:29 limit at which beyond that 29:30 29:30 it's diminishing returns okay you're 29:32 29:32 going to get a slight improvement by 29:33 29:33 improving the 29:34 29:34 the sample but but but not much so it's 29:37 29:37 actually important to try and minimize 29:39 29:39 the sample size 29:40 29:40 by understanding what is it that you 29:42 29:42 want to do 29:46 29:46 thank you noah um if i could pose one 29:50 29:50 more question 29:51 29:51 to you this is also two questions that 29:54 29:54 have come 29:54 29:54 in um asking how did the sampling in the 29:57 29:57 case of the united states go so 29:59 29:59 go so wrong um with reference to florida 30:02 30:02 for an example 30:03 30:03 um and given that in the 2017 election 30:07 30:07 the polls 30:08 30:08 were didn't predict the actual result 30:11 30:11 can 30:11 30:11 people really trust polls 30:15 30:15 okay so a couple of things to clarify 30:18 30:18 so this is one of the 2016 polls 30:20 30:20 probably one of the more reputable 2016 30:22 30:22 polls 30:23 30:23 so it's not that the the polls said 30:27 30:27 um the so you don't interpret this as 30:29 30:29 the poll definitely said that mrs 30:31 30:31 clinton is going to win 30:32 30:32 the what the paul said was that mrs 30:34 30:34 clinton had a 71.4 30:36 30:36 percent chance of winning so definitely 30:37 30:37 higher than trump but 30:39 30:39 uh the trump uh victory wasn't 30:41 30:41 impossible so so in general this is an 30:44 30:44 important thing to 30:45 30:45 kind of keep in mind about statistics uh 30:48 30:48 statistics doesn't give you certainty 30:51 30:51 it basically tries to make the best of 30:53 30:53 uncertainty and kind of minimize 30:55 30:55 uncertainty but it can't 30:57 30:57 um it can't completely get rid of it so 30:59 30:59 for example 31:01 31:01 statistics can't tell you okay if you 31:03 31:03 toss a 31:04 31:04 coin twice uh uh what the outcomes are 31:07 31:07 going to be that you can't solve that 31:09 31:09 problem using statistics and this is 31:10 31:10 something similar 31:11 31:11 so basically another way of describing 31:13 31:13 this result was you toss a coin 31:15 31:15 twice um if you get heads twice it's 31:18 31:18 trump 31:19 31:19 any other result it's it's it's clinton 31:21 31:21 actually trump says trump's odds are 31:23 31:23 like a little bit better than that but 31:24 31:24 but roughly that's what this means so 31:26 31:26 you can't get absolute certainty 31:31 31:31 sure thank you um do you have the time 31:34 31:34 to take one more question new one or 31:36 31:36 shall we leave that 31:38 31:38 yeah let's let's let's do one more 31:40 31:40 question one more sure 31:42 31:42 so we have a question coming in from uh 31:44 31:44 niwal colin berger asking 31:46 31:46 how would you sample for a water survey 31:48 31:48 say for example in the colombo district 31:50 31:50 would it suffice to do a stratified 31:52 31:52 random sampling 31:53 31:53 based on relative populations in each 31:56 31:56 electoral district 31:58 31:58 or do you have to factor in other stuff 31:60 31:60 like ethnicity 32:01 32:01 and religion um so so i think i think 32:05 32:05 the 32:05 32:05 the correct answer is is all of those so 32:08 32:08 um so colombo has i think around 32:11 32:11 something like 15 electoral districts 32:14 32:14 and so if you 32:16 32:16 and just as a sample which has has 32:18 32:18 ninety percent males 32:20 32:20 uh a sample which has a representation 32:22 32:22 from only a handful of electoral 32:24 32:24 districts 32:25 32:25 is is is is not going to work so so 32:28 32:28 you should initially try to make sure 32:31 32:31 that 32:32 32:32 that the sample is random enough so that 32:35 32:35 um 32:35 32:35 so that you have representation across 32:37 32:37 electoralistics 32:38 32:38 so assuming say if samples are done face 32:41 32:41 to face or 32:42 32:42 or by phone you could try and make sure 32:45 32:45 that 32:45 32:45 you're getting enough coverage in each 32:47 32:47 of the each of the electoral districts 32:49 32:49 because 32:50 32:50 say assuming that you know where the the 32:52 32:52 the the 32:54 32:54 sample people live so you could try and 32:56 32:56 make it as random as possible that way 32:59 32:59 but um so what the question um uh the 33:02 33:02 the the question i 33:03 33:03 mentioned uh stratifying stratifying is 33:05 33:05 what you do 33:06 33:06 when you have these samples which are 33:08 33:08 kind of not completely random 33:10 33:10 you you you you try to correct 33:13 33:13 for uh the the differences in in 33:17 33:17 in across different groups by by 33:20 33:20 weighting samples in kind of different 33:21 33:21 ways so for example 33:22 33:22 if you have a sample which is say 33:25 33:25 60 male 40 female you can try and 33:29 33:29 correct 33:30 33:30 uh for that by waiting by sort of over 33:33 33:33 weighting or over sampling the female 33:35 33:35 group to kind of correct for that 33:37 33:37 that mistake sure uh 33:40 33:40 thank you nuan um shall we pause on the 33:43 33:43 questions for now 33:44 33:44 i'll let you move on to your final case 33:46 33:46 study and then we will have a little bit 33:48 33:48 of time at the end of the discussion 33:49 33:49 at the end of the lecture to take a few 33:51 33:51 more questions 33:53 33:53 okay sounds good um so let's let's move 33:56 33:56 on to our last 33:57 33:57 case study okay so so in our second case 34:00 34:00 study 34:01 34:01 um if your if your sample is big enough 34:05 34:05 and it's also properly 34:07 34:07 it's randomly randomly uh collected 34:10 34:10 um it's it's fairly straightforward um 34:13 34:13 you're likely to get a correct answer um 34:16 34:16 correct conclusions with fairly basic 34:19 34:19 statistics 34:20 34:20 so i want to discuss a case where we can 34:23 34:23 use some 34:23 34:23 more powerful data science techniques 34:26 34:26 and that's 34:27 34:27 this predicting elections in real time 34:30 34:30 so what i mean by this 34:31 34:31 is when new results 34:34 34:34 in election come in you try to use those 34:36 34:36 results to try and predict the final 34:38 34:38 result so for example in the current 34:41 34:41 um current u.s election various results 34:43 34:43 are coming in and people are trying to 34:44 34:44 predict like will 34:46 34:46 will will this bill brighten with this 34:48 34:48 state or will trump in this state and so 34:50 34:50 on 34:50 34:50 and so how you do it is you train a 34:52 34:52 model using historical data 34:54 34:54 and you send your new results through 34:57 34:57 that model 34:57 34:57 and that will predict um what the final 34:59 34:59 result would be so in this case i'm not 35:01 35:01 going to discuss the u.s 35:03 35:03 election i'm going to discuss the recent 35:05 35:05 general election in sri lanka where 35:07 35:07 we can use a real-time model to predict 35:09 35:09 how many seats each party is going to 35:11 35:11 get at the end 35:12 35:12 so before we move on to the model itself 35:15 35:15 a quick couple of quick thoughts on data 35:18 35:18 collection and mining 35:19 35:19 especially in areas like machine 35:21 35:21 learning the quality of the machine 35:23 35:23 learning depends on the quality of the 35:25 35:25 data 35:26 35:26 and unfortunately many students of 35:28 35:28 machine learning are not willing to put 35:29 35:29 in the hard 35:30 35:30 work beforehand to make sure that data 35:32 35:32 collection and data mining happens in an 35:34 35:34 efficient way so this is not this is not 35:37 35:37 very exciting it 35:38 35:38 can be tedious and a lot of hard work 35:40 35:40 but it's still very important 35:42 35:42 so in our case there are two sources of 35:44 35:44 data 35:45 35:45 the first source of data are the 35:47 35:47 historical election results 35:50 35:50 and all sri lankan election results are 35:53 35:53 published on the election commission 35:55 35:55 site where you can 35:56 35:56 extract um one problem is that these 35:60 35:60 results are published as pdfs so while 36:02 36:02 they're in a digital form they're not in 36:04 36:04 a form that can be 36:05 36:05 easily analyzed so one pre 36:08 36:08 pre-analysis step is to actually pass 36:10 36:10 these pdfs and convert it into 36:13 36:13 a cbs csv or a json file that can be 36:17 36:17 more easily passed 36:18 36:18 so that that is one of the first steps 36:20 36:20 in this uh if you want to build a 36:22 36:22 a real-time model for for sri lankan 36:24 36:24 elections 36:25 36:25 you don't actually have to do this step 36:27 36:27 because i've already 36:28 36:28 passed and cleaned up the data and you 36:30 36:30 can find that on my 36:32 36:32 github page okay the second source of 36:34 36:34 data that we need for 36:36 36:36 this real-time prediction is the actual 36:38 36:38 results as they come in 36:40 36:40 this is much easier to collect because 36:43 36:43 uh since the last couple of elections 36:45 36:45 the 36:45 36:45 election commission have been publishing 36:47 36:47 these real-time results 36:49 36:49 uh on the official results uh website as 36:52 36:52 they come in so all you need to do is to 36:54 36:54 just go to the site and download the 36:56 36:56 results so you can write a script which 36:58 36:58 which downloads those results 36:59 36:59 automatically okay 37:01 37:01 so so that's the data let's now see how 37:04 37:04 we can build the model 37:05 37:05 and how and we're going to build a 37:07 37:07 machine learning model for this but but 37:08 37:08 let me give a little bit of introduction 37:11 37:11 to machine learning before we go into 37:12 37:12 the model itself 37:14 37:14 so here's some data about house prices 37:17 37:17 on the 37:17 37:17 x-axis you have the square footage of 37:20 37:20 the house and on the y-axis you have the 37:22 37:22 the price of the house so basically what 37:25 37:25 this 37:25 37:25 kind of indicates is that there's a 37:27 37:27 linear relationship between the area of 37:29 37:29 the house and the price of the house 37:31 37:31 by the way these are uh this this is u.s 37:33 37:33 data so the the price is in in dollars 37:37 37:37 uh so if you apply a little level 37:39 37:39 geometry to this you'll see that okay 37:41 37:41 that line is y equals mx plus c 37:44 37:44 uh and or in other words the house price 37:46 37:46 is a constant 37:47 37:47 times the house area plus another 37:49 37:49 constant 37:51 37:51 um more generally what's happening here 37:53 37:53 is we are 37:54 37:54 we are we are defining house price as a 37:57 37:57 function of house area so therefore if 37:58 37:58 you know 37:59 37:59 house area you know house price so 38:03 38:03 a machine learning model uh 38:06 38:06 a model output by machine learning 38:08 38:08 training is actually a fancy word for 38:10 38:10 just a function a machine learning model 38:12 38:12 is actually a function 38:13 38:13 which maps what you know into what you 38:16 38:16 want to know 38:17 38:17 so if you know if you if you have that 38:19 38:19 function and 38:20 38:20 and you know something you can map it 38:22 38:22 into what you want to know or in other 38:23 38:23 words you 38:24 38:24 it's a function which maps data into 38:26 38:26 useful information 38:27 38:27 so so none of these functions are 38:29 38:29 perfect because as you can see that line 38:31 38:31 doesn't fit all the data points there 38:33 38:33 are some 38:34 38:34 uh dots on that chart which are kind of 38:37 38:37 away from the line and so this is the 38:39 38:39 error so all machine learning models 38:41 38:41 also have 38:42 38:42 associated error so so how does this 38:45 38:45 work in practice 38:46 38:46 so what you do is um you write some 38:50 38:50 code which takes in uh the the the x and 38:54 38:54 y variables 38:55 38:55 and you can write a a 38:59 38:59 a function which in in turn takes the x 39:01 39:01 and y variables and outputs the 39:03 39:03 parameters of the model in the 39:05 39:05 case of the simple linear model it's the 39:08 39:08 the m and the c in the y equals m x plus 39:11 39:11 c 39:12 39:12 um now many introductory machine 39:15 39:15 learning courses will start with linear 39:17 39:17 regression because linear regression is 39:19 39:19 probably the simplest 39:20 39:20 uh kind of machine learning algorithm 39:23 39:23 but surprisingly it works 39:25 39:25 in many cases so for this real-time 39:27 39:27 prediction i didn't i 39:28 39:28 i used a variation on on on linear 39:31 39:31 regression 39:32 39:32 but even basic linear regression would 39:34 39:34 work quite well 39:35 39:35 and in our example the x and y's uh the 39:38 39:38 y's are what you want to know 39:40 39:40 say for example you can build a model to 39:42 39:42 find out how many seats there's a pp are 39:44 39:44 going to get in the columbia district 39:46 39:46 and the x are the the results 39:49 39:49 that you feed into this model so so many 39:52 39:52 people ask 39:53 39:53 okay why did i use linear regression in 39:55 39:55 this case why can't you use a more 39:58 39:58 more sophisticated model let's say using 39:60 40:00 deep learning 40:01 40:01 so the answer is that you can and if you 40:03 40:03 use a more sophisticated model you could 40:05 40:05 get a slight improvement but not much 40:09 40:09 and the reason for that is more 40:11 40:11 sophisticated models like 40:12 40:12 like very complex neural networks uh uh 40:15 40:15 work uh give a uh kind of incremental 40:18 40:18 benefit 40:19 40:19 uh only when you're working on kind of 40:21 40:21 very large sets of data 40:23 40:23 so a single sri lankan election would 40:26 40:26 have a few kilo 40:27 40:27 bytes actually maybe about 30 or 40 40:29 40:29 kilobytes of data 40:30 40:30 so our entire history of elections plus 40:33 40:33 various other data points you might want 40:34 40:34 to use like demographics and so on 40:37 40:37 all the data will be less than a 40:38 40:38 megabyte and so linear regression would 40:40 40:40 work perfectly fine 40:42 40:42 okay let's see the model of our output 40:45 40:45 so you 40:45 40:45 probably remember this visualization 40:48 40:48 from 538 40:50 40:50 so here's where 538 predicted uh 10 40:53 40:53 chance of a trump win and a 98 89 40:56 40:56 chance of a biden win um each of these 40:59 40:59 circles uh 41:00 41:00 represent a possible scenario a possible 41:04 41:04 future 41:05 41:05 outcome of the election so for example 41:08 41:08 uh this 41:09 41:09 dot represents this outcome 41:12 41:12 so this is exactly how our model works 41:16 41:16 as well and the output is very similar 41:18 41:18 so for example 41:19 41:19 um when uh for the model which predicts 41:22 41:22 those 41:23 41:23 slpp seats in the columnar district it 41:25 41:25 doesn't output one value 41:27 41:27 it does simulations and produces a set 41:30 41:30 of different values which are all kind 41:31 41:31 of equally possible in the future 41:34 41:34 so here's an example of 20 simulation 41:37 41:37 outputs 41:38 41:38 and so what we do here is we sort the 20 41:42 41:42 and we pick the 18 in the middle 41:46 41:46 and if you think about it 18 out of 20 41:48 41:48 is 90 41:49 41:49 and this gives us a 90 percent 41:51 41:51 confidence 41:52 41:52 uh of the number of seats in of the srpp 41:56 41:56 are going to get in colombo 41:57 41:57 so basically what it's saying is okay 41:58 41:58 there's a 90 chance that it's going to 42:00 42:00 get between 11 and 13 seats or 42:02 42:02 alternatively 12 plus or minus one 42:05 42:05 so in my model i actually use 95 42:08 42:08 in that case instead of 20 you do 40 and 42:11 42:11 pick 42:11 42:11 the 38 in the middle so 38 out of 40 is 42:14 42:14 is 98 42:17 42:17 okay so let's see how you visualize this 42:20 42:20 so so uh visualization and data 42:23 42:23 presentation is sometimes considered 42:25 42:25 less 42:25 42:25 science and more or not and sometimes 42:28 42:28 gets 42:29 42:29 not enough attention as it should uh i'm 42:32 42:32 not expert in the area but i'm going to 42:34 42:34 describe how i attempted to visualize 42:36 42:36 the results of 42:37 42:37 the the real-time models that i built um 42:40 42:40 the the complete result looks something 42:42 42:42 like this and i'd like to go 42:44 42:44 through some of the principles that i 42:46 42:46 used when i designed this 42:48 42:48 so the most important principle 42:51 42:51 in designing visualization is probably 42:53 42:53 minimalism 42:55 42:55 in that you don't want to put too much 42:56 42:56 information which is not necessary so 42:58 42:58 for example 42:59 42:59 on the left hand side you see the 43:01 43:01 results of every single party 43:03 43:03 um being reported and actually most 43:06 43:06 people probably just want to know what 43:08 43:08 the top three or the top four 43:10 43:10 are another example is if you take the 43:12 43:12 percentage vote 43:13 43:13 is it actually necessary to have the 43:15 43:15 percentage more to two decimal places 43:17 43:17 um 22 sounds everything that you want 43:20 43:20 you don't need to know that it's sort of 43:22 43:22 22.48 43:23 43:23 so that's minimalism trying to 43:25 43:25 minimalize the design 43:27 43:27 paradoxically the other concept is 43:29 43:29 maximism 43:30 43:30 so what i mean by that is you have a 43:32 43:32 limited space to show something 43:35 43:35 and so you should try to make the best 43:36 43:36 of that space so the map on the left 43:39 43:39 has the winners of the the recent 43:41 43:41 general election 43:42 43:42 the map on the right has the same 43:45 43:45 results though though for a subset 43:47 43:47 because not all the results have been 43:49 43:49 released at that point 43:50 43:50 but it has more information in that it 43:52 43:52 uses color shades um 43:54 43:54 to say how much what's the the winning 43:57 43:57 party got so from from the 43:59 43:59 image on the right you can say that the 44:02 44:02 um the slpp's performance in new aurelia 44:04 44:04 for example 44:05 44:05 uh was very different from monarch 44:07 44:07 [Music] 44:10 44:10 so the the last bit was the actual 44:12 44:12 projection results as 44:14 44:14 uh itself um you saw that the model said 44:17 44:17 that the slpp was going to get 11 to 13 44:20 44:20 seats in colombo and so what that means 44:22 44:22 is the model thinks that 44:24 44:24 uh the the the srpp is going to get at 44:26 44:26 least 11 44:28 44:28 seeds so this was represented as kind of 44:30 44:30 11 circles in 44:32 44:32 in the sleeping color similarly there 44:34 44:34 was another model which predicted the 44:36 44:36 number for the sjp and it predicted six 44:39 44:39 to eight and so you've got the 44:41 44:41 the the six green circles and then the 44:44 44:44 remaining circles 44:45 44:45 represent for grab seats and so you do 44:47 44:47 the same thing for all the districts 44:49 44:49 and the final results look something 44:51 44:51 like this um 44:52 44:52 and so you have the colors for the 44:54 44:54 different parties who have guaranteed 44:56 44:56 seats 44:56 44:56 and then the remaining seats which are 44:58 44:58 still in play 44:60 44:60 are sort of empty circles and so finally 45:02 45:02 the final 45:03 45:03 final result looks something like this 45:05 45:05 what you see on the left are the 45:07 45:07 official finalized seats and what you 45:09 45:09 see on the right are the projections 45:11 45:11 about 45:11 45:11 two thirds of the way into the elections 45:13 45:13 so basically it was 45:15 45:15 very clear from the projection but the 45:17 45:17 final result was if you think about it 45:18 45:18 this is very similar to what the final 45:20 45:20 result 45:21 45:21 actually was okay so that's that's all i 45:23 45:23 had 45:24 45:24 um before i kind of end uh there are two 45:27 45:27 people that i should thank 45:29 45:29 uh one is my sister devine who's 45:31 45:31 actually a data scientist 45:32 45:32 um unlike me uh and my wife kana who 45:35 45:35 gave a lot of good feedback on these 45:36 45:36 slides 45:37 45:37 uh and and some suggestions as well um 45:40 45:40 also if you want to learn more details 45:42 45:42 about all the things that we discussed 45:44 45:44 today you can follow my articles as a 45:45 45:45 medium 45:46 45:46 and you can also follow me on twitter 45:49 45:49 which has 45:50 45:50 has has kind of similar stuff so good 45:53 45:53 so let's move on to q a 45:57 45:57 great thank you so much noah um so for 45:60 45:60 the q a section 46:01 46:01 we have uh quite a few more questions 46:04 46:04 that have come in 46:04 46:04 on the q and a chat box uh but if you 46:08 46:08 would also like to 46:09 46:09 ask anyone your question directly 46:13 46:13 be you have that option just raise your 46:16 46:16 hand so to raise your hand there's just 46:17 46:17 a little 46:18 46:18 hand icon at the bottom of his screen uh 46:20 46:20 click on that and then i will be able to 46:22 46:22 call 46:22 46:22 on you and you will be able to answer 46:24 46:24 your mic uh or turn your video on and 46:27 46:27 ask the question directly 46:29 46:29 um but while people decide whether 46:32 46:32 they're 46:33 46:33 going to uh make use of that option we 46:36 46:36 can start with some questions that have 46:38 46:38 come in 46:38 46:38 on the chat um so we have one question 46:42 46:42 in 46:43 46:43 um from shamindra kulamanage 46:46 46:46 asking in exploring how to create 46:48 46:48 interactive digital maps 46:50 46:50 i realized the need for geo json files 46:54 46:54 for 46:54 46:54 admin districts and polling division 46:56 46:56 level for elections 46:58 46:58 um is there an open database for that 47:01 47:01 sort of information in sri lanka 47:04 47:04 the the correct answer is yes the the 47:07 47:07 department of census and statistics has 47:09 47:09 an open uh data set 47:11 47:11 unfortunately it's not very clearly 47:13 47:13 visible um 47:14 47:14 so if um if the question i can 47:17 47:17 just send me some contact details i 47:19 47:19 could send more or actually i might 47:21 47:21 actually post it 47:22 47:22 on this talk page more details on that 47:25 47:25 i'll just make a note of that 47:27 47:27 great thank you um i think we have 47:31 47:31 one person raising their hand uh 47:36 47:36 puree or ms 47:39 47:39 um let me i've asked you to unmute if 47:42 47:42 you would like to ask your question 47:51 47:51 um i think 47:53 47:53 the question is done one second i'm 47:57 47:57 asking 47:58 47:58 yeah hello please 48:01 48:01 yes please keep your question brief uh 48:04 48:04 i'm in this guru calling from japan 48:06 48:06 thank you very much for interesting 48:08 48:08 presentation i'm just wondering whether 48:10 48:10 you are engaged in some kind of 48:12 48:12 macroeconometrics modeling for sri lanka 48:17 48:17 um actually no i i do this mostly as a 48:21 48:21 hobby 48:21 48:21 uh my my day job is in in in computer 48:24 48:24 science so i work in ai 48:26 48:26 this is this is more of a kind of a side 48:28 48:28 interest 48:29 48:29 i should do what you recommend oh okay 48:32 48:32 i'm not recommending i'm also doing this 48:35 48:35 kind of work i'm the 48:37 48:37 forecasting so i thought maybe if you 48:39 48:39 are doing maybe 48:41 48:41 i can share some of my work later on 48:44 48:44 yeah 48:45 48:45 yeah i would have i'll be happy to 48:46 48:46 connect thanks yeah thank you 48:48 48:48 thank you um so we have another question 48:53 48:53 um in the chat asking if there is a way 48:56 48:56 to get our media to have more election 48:58 48:58 coverage that's based on data science 49:03 49:03 um so yeah so that that that's a good 49:05 49:05 question 49:06 49:06 um and uh i think in general we 49:10 49:10 uh that that probably applies to uh 49:12 49:12 other areas 49:13 49:13 as well not not not just elections um i 49:16 49:16 i think 49:17 49:17 at the moment the the the the 49:20 49:20 responsibility is probably on the side 49:22 49:22 of 49:22 49:22 data scientists uh uh i'm not a data 49:26 49:26 scientist myself but okay let me wear 49:27 49:27 that 49:28 49:28 for a moment i think we need to we need 49:30 49:30 to uh take more responsibility and on 49:33 49:33 kind of demonstrating 49:34 49:34 uh the value of data science outside our 49:37 49:37 sort of inner circle a little bit more 49:39 49:39 so at the moment it's seen as a sort of 49:40 49:40 a very technical field 49:42 49:42 um it's not the sort of thing where you 49:45 49:45 might have a sort of a 49:46 49:46 data scientist speaking on a popular tv 49:48 49:48 show as you might see 49:50 49:50 uh say uh uh economist or a political 49:54 49:54 scientist on a popular show so i think i 49:56 49:56 think i think data scientists need to 49:58 49:58 get out a little bit more and and and 50:00 50:00 promote the subject also a little bit 50:02 50:02 more so i think i think that's probably 50:03 50:03 the 50:03 50:03 the first thing to do 50:07 50:07 um so i think this is something the next 50:09 50:09 question is something you have touched 50:10 50:10 on briefly during the lecture 50:13 50:13 we have a question in asking how do you 50:15 50:15 increase the representativeness of a 50:17 50:17 sample for example if you want to survey 50:20 50:20 a thousand people 50:21 50:21 how do you find a thousand who 50:22 50:22 constitute a representative sample 50:26 50:26 and then kind of tagging on to that um 50:29 50:29 can there be a bias for people who would 50:31 50:31 hide their answers because 50:33 50:33 of social pressures the example given is 50:35 50:35 if your 50:36 50:36 preference is not seen as to be walk 50:39 50:39 enough 50:41 50:41 yeah i think i think definitely in the 50:42 50:42 case of the the u.s election that that 50:45 50:45 looks kind of very true um it was 50:47 50:47 definitely true in 2016 it looks like 50:50 50:50 it's too true in 2020 as well 50:53 50:53 um so that i think that's that that goes 50:56 50:56 beyond elections it might be sort of a 50:58 50:58 cultural problem where you have 50:60 50:60 a country where half the country are not 51:02 51:02 willing to speak to the other half of 51:04 51:04 the country and 51:04 51:04 so there is this real communication gap 51:07 51:07 and if you think about it 51:08 51:08 things like uh polls uh uh tend to be 51:12 51:12 done by uh by by 51:15 51:15 uh sorry intellectuals statisticians 51:18 51:18 that sort of person and 51:19 51:19 there is a probability that those people 51:21 51:21 are more concentrated on the political 51:24 51:24 left 51:25 51:25 and so there's that unconscious bias 51:27 51:27 going in as well so that might be kind 51:29 51:29 of another problem 51:33 51:33 um so we have a few more questions i 51:36 51:36 think we can maybe take 51:37 51:37 two more before we have to wrap up this 51:40 51:40 session 51:41 51:41 um so we have one question coming in 51:42 51:42 asking um is there a software you 51:44 51:44 recommend for 51:46 51:46 use for real-time projections 51:49 51:49 um so all the the projections 51:52 51:52 the the only uh piece of software per se 51:55 51:55 that i used was was 51:56 51:56 was the python programming language um 51:59 51:59 everything you 51:60 51:60 need is is actually there from 52:03 52:03 from scraping the data of websites and 52:05 52:05 passing the pdfs 52:07 52:07 to um to actually training the models 52:09 52:09 and also creating those visualizations 52:11 52:11 those are also 52:12 52:12 um so for some so so i i just use python 52:16 52:16 throughout because 52:17 52:17 because i'm quite familiar with that and 52:19 52:19 probably because i'm a computer 52:21 52:21 scientist as well like 52:22 52:22 coding comes naturally but for some 52:24 52:24 other pieces there are other pieces of 52:26 52:26 software which are better 52:27 52:27 so for example for the last step the 52:29 52:29 visualization 52:31 52:31 um products like d3 52:34 52:34 uh using javascript are probably better 52:36 52:36 than using using python 52:38 52:38 and then for things like scraping and 52:41 52:41 mining data there are 52:42 52:42 for example customized products for 52:45 52:45 extracting tables from pdfs things like 52:47 52:47 that 52:48 52:48 um so so yeah so so you can use one for 52:50 52:50 everything but you can also do the 52:52 52:52 opposite and use specialized 52:53 52:53 uh software for each of the different 52:55 52:55 things 52:58 52:58 great thank you um so i'm going to 53:02 53:02 i'm going to post two more questions at 53:03 53:03 your new one sorry uh 53:05 53:05 we have another question that's asking 53:07 53:07 how do you get data 53:08 53:08 using facebook or twitter 53:11 53:11 um so there are a couple of different 53:14 53:14 ways 53:15 53:15 um so one is um if you if you you can 53:19 53:19 run pose of your own 53:20 53:20 and in which case you have access to all 53:22 53:22 the data 53:24 53:24 um you can you can especially if you are 53:27 53:27 someone like academic you can go into a 53:29 53:29 partnership 53:29 53:29 with uh with facebook and twitter and so 53:32 53:32 you can you might have access to online 53:34 53:34 anonymized data from inside which normal 53:37 53:37 people don't have access 53:38 53:38 i know facebook recently just revamped 53:41 53:41 their program 53:42 53:42 in by which they collaborate and 53:44 53:44 collaborate with academics and so it has 53:46 53:46 got a lot 53:46 53:46 a lot easier so that's that's that's 53:48 53:48 another way and like a third way is this 53:51 53:51 is easier to do on twitter than on 53:53 53:53 facebook where you can just go and kind 53:55 53:55 of scrape the data off the website 53:56 53:56 itself 53:57 53:57 uh the same way you would scrape a any 54:00 54:00 website but that can be tricky 54:02 54:02 um those those platforms could block you 54:04 54:04 especially if it violates their terms 54:06 54:06 so yeah there are a couple of different 54:07 54:07 ways you can collect the data 54:09 54:09 sure and and the final question for 54:12 54:12 today is 54:13 54:13 um as someone who wants to get into data 54:15 54:15 analytics 54:16 54:16 are there any courses you recommend 54:19 54:19 um so uh so there are quite a lot of 54:22 54:22 questions so if you look at 54:23 54:23 places like coursera and udemy or just 54:27 54:27 kind of look at youtube you have you 54:29 54:29 have a whole lot of videos on on the 54:31 54:31 basics of of data analytics 54:34 54:34 but i would recommend before finding 54:37 54:37 courses find an 54:38 54:38 interesting problem find the interesting 54:40 54:40 data analytics problem that you're kind 54:41 54:41 of really interested in 54:43 54:43 because the problem is you could take a 54:45 54:45 course um 54:46 54:46 but in in six months time you could 54:48 54:48 forget everything you learned 54:49 54:49 unless you actually applied so i would 54:51 54:51 say start with the problem and then 54:53 54:53 see okay what are the tools and 54:55 54:55 knowledge that you need to solve that 54:57 54:57 problem 54:59 54:59 great thank you very much no one um so 55:02 55:02 unfortunately i think our time is up now 55:05 55:05 um we have had about 70 people roughly 55:07 55:07 sitting in on this zoom call 55:09 55:09 um with few variations as people join 55:12 55:12 and leave 55:12 55:12 um so i would like to thank uh yunuan 55:15 55:15 for joining us today and for preparing 55:17 55:17 all of this and for what was a very 55:19 55:19 interesting 55:20 55:20 lecture and discussion session um i 55:22 55:22 would also like to join everyone 55:24 55:24 i would also like to thank everyone who 55:26 55:26 has joined us today 55:27 55:27 um on zoom and on our facebook platforms 55:31 55:31 uh this event was a part of the think 55:33 55:33 series which was co-hosted by the 55:34 55:34 advocate institute and echelon magazine 55:37 55:37 and on behalf of the co-host i would 55:39 55:39 like to thank our streaming partners 55:40 55:40 echelon magazine economy next blue chip 55:43 55:43 training and consulting 55:44 55:44 newshub.lka newsup.org english 55:47 55:47 business news citizen daughter k economy 55:50 55:50 and business sri lanka 55:52 55:52 daily ft the morning the sunday morning 55:55 55:55 business 55:55 55:55 the night watchman society code 2.0 and 55:58 55:58 punjabi 55:59 55:59 the think series will continue so please 56:01 56:01 follow the advocate institute and 56:03 56:03 echelon magazine and our social media to 56:05 56:05 stay updated 56:06 56:06 thank you very much thank you