How Recruiters Are Using Artificial Intelligence w/ Dr. Lindsey Zuloaga @HireVue (Episode 30) #DataTalk

Listen to the podcast:

Every week, we talk about important data and analytics topics with data science leaders from around the world on Facebook Live.  You can subscribe to the DataTalk podcast on iTunes, Google Play, Stitcher, SoundCloud and Spotify.

This data science video and podcast series is part of Experian’s effort to help people understand how data-powered decisions can help organizations develop innovative solutions and drive more business.

To keep up with upcoming events, join our Data Science Community on Facebook or check out the archive of recent data science videos. To suggest future data science topics or guests, please contact Mike Delgado.

In our recent #DataTalk, we chatted with Dr. Lindsey Zuloaga from HireVue about how recruitment teams are using artificial intelligence to help them find and hire the right candidates.

Check out the full interview:

Mike Delgado: Hey, friends. Welcome to our weekly #DataTalk, a show where we talk to data science leaders from around the world. Today, we’re talking about how recruiters are using artificial intelligence to find the right candidate. Super excited to chat with Dr. Lindsey Zuloaga.

She is the Director of Data Science at HireVue. She received her Bachelor of Science in applied physics from The University of Utah and then went on to earn her Ph.D. in applied physics from Rice University.
For those who are watching live, feel free to type in any questions or comments you have about recruitment and AI. We’re excited to have Lindsey with us. Lindsey, I always love to start these chats asking our guests to tell us a little bit about your journey that led you into data science.

Lindsey Zuloaga: I was always good at math as a kid, but I never really understood what it was for until I took physics. I had a great teacher my senior year of high school who made physics fun and opened up this whole world to me where I felt like I really connected the two. This is a math test for, you can yield so I am the first person in my family to go to college. I was feeling like I didn’t know if I was going to be smart enough to major in physics. Just because I was good at it in high school doesn’t mean I could major in it.

I just kept chugging along and really enjoying it, just decided the worst thing that could happen is that I’ll fail and have to choose something else. During undergrad, I did some summer research projects and I got into research and then applied to grad school after undergrad. I did a master’s in Ph.D. for six years at Rice, and then I did a year of a postdoc in Germany. I was in the academic world for a long time, and when you’re in that world it’s just a bubble. Like all anybody cares about is that world, so you don’t know what’s outside of that.

I was interested in what we called the real world and maybe venturing outside of academia. I had been doing well in academia, and that’s a really competitive space. So I felt like I’m going to be fine, I’m just going to go get a job and I could do a lot of things. I was optimistic when I started my job search, but I was quickly disillusioned. This happens to a lot of people transitioning industries.

I thought my résumé speaks for itself, and I just went out and applied for jobs online, through applicant tracking systems. Many people probably experienced this before, where you have to upload your résumé and your cover letter. You spend a lot of time, especially on your cover letter. You don’t want it to sound dry. You want it to be personalized for that job. So you upload that and then it just gets converted into plain text, and then you have to re-enter everything a lot of times and then you press “Submit.”

I’ll often never hear anything, and your information goes into a black hole and you feel like no human beings are ever seeing your résumé. It’s a pretty awful feeling, and I’ve since learned many people in my shoes went through a similar thing. I think a big thing for me is I did underestimate the value of networking. I just thought my résumé speaks for itself. You really do need to get to know people and talk to people about your ideas and present yourself as a person, like your excitement and your interest, your passions.

I always felt pretty sure that if I could just talk to people, I think I’d be doing a lot better. Just a résumé is a bad representation of a person. I ended up getting a job as a data scientist on a business analytics team, and that’s how I started.
When I first heard about HireVue, what I was interested in was it tied into my own experience because I feel like this hiring space is very broken. I felt like once I did get a job, I was fine and now I have opportunities open to me and more. During the job search, you start to question your value and that experience for job applicants.

It’s very inefficient for companies too because they miss out on a lot of good talent that these applicant tracking systems just filter through to try to find certain keywords. Often companies are going to hire someone they know anyway, but they have to post the job, so you’re just wasting your time.

At HireVue, we have a video interviewing platform. Our whole idea is based on the résumé is not a great representation of a person. So you should be able to record yourself talking, answer some questions, and express who you are and what you care about, rather than just a résumé. That’s been our history, building a video interviewing platform. Now we’ve also added artificial intelligence on top of that. I can talk about that more, but that’s my story.

Mike Delgado: So many people I’ve talked to have also felt the same way. You go to Indeed.com, LinkedIn, or Yahoo Jobs and you see all these job listings and you apply. You go through that crazy process of submitting your résumé, and like you said, you have to type it all many ways. Then try to craft the right personalized cover letter and then you never hear from them again, or maybe you get an email two months later saying, “Thank you so much for applying, but we found somebody else.”

What I think is cool about HireVue is your approach is to make it more personal because, like you said, a candidate is more than just what’s on the résumé. There’s a whole person behind that, and oftentimes it’s through video where you can see personality traits and get a feel for if this is the right kind of person. Is this person going to bring the right energy? Can we see this person on our team? You can see that through video.

Lindsey Zuloaga: Yeah, and a lot of people have experienced going through all these early stages of hiring someone. Then from the hiring side, you sit down to interview them and very quickly you can sense that it’s not the right person. It’s like dating. People can talk online and send messages on and on and on, but it can often be a waste of time because when you meet someone, you get a sense of who they are very quickly. It’s better to do that early on and see if it’s a fit or it’s a waste of both parties’ time.

Mike Delgado: No doubt. Tell me a little bit about how AI is being used in these video recordings of candidates looking for jobs.

Lindsey Zuloaga: A big motivation is we have a lot of companies that have high volume, and they get a lot of applicants. It’s hard to know where to start with looking at applicants, so our thinking is we can see from video interviews from the past which candidates ended up being really good at their job and different things, depending on the job. It could be sales numbers. If it’s a high turnover, it could be retention rate.

So we have a performance metric that we try to predict. We train on those previous interviews, and then we apply our models to incoming people. We look at all the words people said and types of subjects they talked about, facial expressions they made, and other audio features like tone of voice and things like that.

A lot of times when people learn about this, they ask me how we decide what’s important for a flight attendant or some other position. To be clear, we don’t make that decision. The data does. But for a flight attendant, for example, what would often come out of a model would be flight attendants who smile and use calm language or something like that. They end up being good at their job. The algorithm is looking for people who show promise to perform well in the position, and then it helps companies sort their candidates in a smart way so hopefully the more promising people get seen first.

A lot of times the best candidates are going to be off the market pretty quickly. For companies, it’s very valuable to not miss out on that talent. For people, it’s valuable to be evaluated on your qualities and how you express yourself and if that’s a good fit for that position. Another thing we talk about a lot, and I think one of the first questions that we get, is about bias. There’s a lot in the media about artificial intelligence being racist and sexist.

It’s something we’re very concerned about and very on top of because as much as machine learning algorithms can mimic human prejudices, they can also help us overcome them. We can control what goes into our algorithm and essentially make it blind to these things, and sometimes that’s harder than people think. Obviously, if I wanted to talk about who gets a loan and who doesn’t get a loan, it would seem fair to not put race as a feature into that model.

You could end up having a model that’s racist and you didn’t even know that somehow race leaked into your model. You have to be very careful about those things. Then in our space there’s a lot of things like that with age, race, gender. There can be things that leak through in the way you speak; women speak faster a lot of times we’ve seen. I’m not making a huge blanket statement.

That is just something I’ve seen come through as a feature.

If there’s any bias in the data, which often there is, and that depends on company, and that depends on country, the algorithm can find things to differentiate people and become biased itself. A big part of what we do is check our algorithms and making sure they don’t treat different groups differently. That’s called adverse impact or disparate impact between groups.

Then if we remove features that are causing those problems … So if the model is still not blind to gender, for example, then we may need to remove certain inputs to basically blind it. We have a pro-diversity message, and we think that what we do helps people from underrepresented classes be seen more than they might have previously.

Mike Delgado: That’s amazing and awesome to hear the work you guys are doing to help create an unbiased AI and keeping tabs on that. Can you talk a little bit about whether it’s primarily recorded video that’s being analyzed or is there also a way to do live video interviews?

Lindsey Zuloaga: Yeah, we do that as well. So often, depending on what point in the funnel candidates are at, a lot of times what we call an on-demand interview, which is asynchronous, is ideal for earlier in the funnel. It creates an environment where there’s no scheduling involved and people can take the interview on their own time whenever it’s convenient for them. Then it can be watched by many people and reviewed by many people on their own time.

Sometimes in-person interviews depend on what mood the interviewer is in. So everyone might not be treated the same and have the same circumstances so you get to compare people. Also, if you’re interviewed by many different people, you might say different things. Sometimes you did better or worse, but everyone watches the same interview. All the reviewers of the interview watch the same interview; all the candidates were asked the same questions. Then further along in the funnel, obviously humans do make the final decision. That’s why interviewing is important.

We do have a live interview platform as well for doing a video chat–style interview. A lot of companies, depending on what their funnel’s like, they’ll fly people in. With using our platform, we’ve had customers who reported to us that they have had to fly far less people in because they can get the sense of who someone is before that [inaudible 00:15:47] and be a little more selective.

Mike Delgado: That saves the company lots of money.

Lindsey Zuloaga: Yeah, exactly. Also time. Just going through money with the time that recruiters take to go through résumés, which is just a horrible experience. I’ve gone through it myself, fairly recently, where you have tons of résumés and you’re just like, “I can’t tell the difference. They’re all so similar.” Even for us, hiring a data science person … We have a data science analyst role that we filled recently. We utilized asking technical questions within the video interview, which I found really useful.

Like just asking people to explain machine learning model in simple terms, or how they would explain deep learning to your grandma or something like that. We can give people coding challenges and see how they code in different, in the language of your choice. We have a lot of clients that use that as well for software developers, things like that.

Mike Delgado: Lindsey, we had a question here from Alex, who’s watching. He says, “Does the algorithm account for any certain terms or words the candidate may say that immediately flags them as not a potential fit?”

Lindsey Zuloaga: No. Like I said, we don’t define any words as bad or good going into it. It’s all just about patterns. It’s all about patterns that were seen in previous top performers in the job or bottom performers in the job. Again, I think a lot of people get a little freaked out about the idea that maybe you could say one word and it ruins your chances.

The thing to remember is that there’s thousands and thousands of these features going into our model so that the data is so rich. It’s not going to be just one little thing that makes or breaks the candidate, but it’s a whole ensemble of things that they do.

Mike Delgado: Tell us about when someone submits their interview for the first time and the recruiters are using HireVue for the technology and the AI. What sort of data is HireVue giving recruiters to help them make decisions? Maybe that’s not even the right word to use, but what type of data is being presented to the recruiter about the person?

Lindsey Zuloaga: It depends on what level and the company a person’s in, but most people have, if they’re set out to review many video interviews, they will just be sorted for them in a smart way. For some customers and some users, we have banding, like green, orange, red score bands. Anybody at that level of detail depends on what the customer wants and the admin rights of the user, how much detail they get behind the score that we assign.

We end up on the back end. What we do is a percentile score to grade people zero to 100, but that’s usually not what most of our users see in the user interface.

Mike Delgado: I was watching a YouTube video about video recruitment. I’m not sure of the company, but I think the produce is in beta. Basically, what it was doing was helping to determine if the candidate was lying. I’m curious if that is part of the technology as well.

Lindsey Zuloaga: Yeah, it’s an interesting thing that we talked about a little. Could we do this? I think that this technology could do that. I could see that it could be a little tricky in a job interview setting with different people being nervous and maybe acting. I guess that’s a psychology question. There are some interesting things we’ve played around a little bit with, like blood flush to your face. Apparently that can be a lie tell, but in order to do that really accurately you need high-quality video enough that you can see that.

That’s somewhat of a limitation to be able to do something that sensitive. Everybody takes job interviews on different devices, and different countries, their internet might be different, so the quality of the video might be a limitation to do that across the board. It’s an interesting idea obviously for other applications or even a job interview if you want to know if this person’s telling the truth or not, but we don’t do anything directly with that. It’s something I suppose could come through in interviews — like all the good salesmen or good liars — and maybe you’re hired if you’re actually good at it.

Mike Delgado: Tell me about the personality types, because often when someone’s trying to … Like you said, they’re looking at résumés that look the same, looks like these people have the same skill set, résumés maybe look identical, they went to identical schools, similar programs. Then they submit the interview and you’re trying to get a sense of if this person really knows what they’re talking about in terms of the questions you’re asking. Then also personality fit. Can you tell us a little bit about how HireVue determines personality types or what type of data is given to the recruiter?

Lindsey Zuloaga: We don’t do a lot on that. We have in our algorithms … There can be different clusters of people who might have been good at the job, so there’s not just one type of person who’s good at it. We can see that if it comes through in the training data, so a limitation is if the company is already very homogeneous, or as far as how they think, then the algorithm can recommend more people like what they have.

That’s something we try to give feedback on when we see training data. This is the demographic makeup of this company or maybe they are very sexist or that’s some feedback that we might want to give them.

We don’t have a lot of good examples, and sometimes it’s just bad luck. There are very few women in engineering. We don’t have very many good examples of how top-performing women may behave as opposed to men. It’s something we’re always trying to get — diversity of that data — but a very interesting question that I think you’re getting at is the team fit and how a diversity of personalities and ideas is important.

That’s something we’re interested in going forward. Can we look at a team of people and say these are the kinds of personalities that work the best together and so this team actually needs someone like this? And this team already has a lot of highly focused technical people, and what it needs is a more creative person. That’s something we’re interested in.

Mike Delgado: That’s cool. Does pulling in all this résumé data and employee data from companies help analyze who would be the best candidates? Are you also looking at job history, like this person tends to jump around quite a bit? So in the video interview, they may do an outstanding job, but there’s also a risk factor, like this person might not be in the role for very long.

Lindsey Zuloaga: Yeah, but we don’t actually incorporate that data. What kind of data you can get is interesting and controversial. You’re looking at something like job history, or some companies will even go out and just mine the whole web for information on this person. Scrape all the social media sites and stuff like that. We pride ourselves on just judging right now this video interview. That is it. You’re coming in with a clean slate, and I think it’s risky to scrape the web for information on people because you’re never really sure this if the web data is accurate.

A lot of things I heard on Facebook, like drinking, doesn’t correlate with job performance, but drug use and bigotry do or things like that. But, yeah, we keep it all within that video interview. Part of that is giving people an equal shot because some people don’t have a very big social media presence, or they have their own story as to why they got where they are or things they’ve done in the past, and those kind of details may come up later in the funnel and are left up to human judgment.

Mike Delgado: Tell me about your advice for the job seeker who is about to do their video interview and they’ve just listened to you describe all the technology, all the data science behind it all, and now they’re freaked out even more, not doing video interview. What’s your advice for them?

Lindsey Zuloaga: Even when I did it the first time, there’s a little bit of awkwardness. I think there’s awkwardness even just with video chatting, and that’s become less awkward as we do it more. When you’re recording yourself answering a question and there’s not a person there giving you feedback, you feel a little bit like you don’t know how you did because no one was nodding or smiling or saying they understood.

Some people like that, but one thing I like to remind people of is that this usually is not replacing an in-person interview. This is not what you’re getting instead of an interview. It’s what you’re getting instead of just your résumé.

It’s pretty early in the funnel. A lot of people feel they would rather have an in-person interview, and it’s like, “You’ve got to earn that first. You’re at the beginning stages.” They need to know that a lot of people do feel a little awkward when they do it at first, and that’s really normal.

Just to behave like you’re talking to a person. As far as being worried about the AI judging you, we build different algorithms for different types of positions and for different companies, so there’s no way to game it.

That’s a big thing with résumés. People will hide a bunch of text in the back of the résumé or get the keywords right, and there’s nothing like that. You’re going to say a certain word over and over again and that won’t help you anyway because eventually a human will watch it and see that. You really just want to be yourself and don’t worry about AI judging you any more than you worry about humans judging you about the gaming thing.

A funny thing we say when people ask if someone could fake it or lie is, “If they’re good enough at faking it.” That’s the rule with humans too. If you’re a good enough actor and you’re pretending that you’re social, then I guess you can fake it in the video interview.

I encourage people to think of an AI judging you very similarly to a human judging. It’s a judge of some sort, and humans are pretty inconsistent and biased, so there’s a lot of reasons to be happy about an AI judging you and hope that that could give you an advantage. I’ve heard people say some scary things, like, “How does anybody stand a chance against the AI?” We still need to hire people. It’s not like no one is good enough. First we try to emulate what a human judge would do and take out some of the bias.

Definitely nothing to be afraid of in general. Treat it like an interview you’re going to, dress well, and present yourself well. Make sure the lighting’s good, don’t have a big mess in the background, you got an impression on people. That’s it.

Mike Delgado: That is good advice. Be real and be authentic; be who you are. It’s funny. One of my friends auditioned for an MTV show. It was like a road trip show back in the day. Everyone had to submit videos to apply because MTV wanted to see what this person looked like and their personality.

He was a cartoonist. He was out in his backyard doing his video, just saying why he would be a good fit on the road trip. He was showing his artwork in the background. It was all comic book stuff that he had drawn. All of a sudden this gust of wind came, and it blew all his art off and he freaked out on camera. Just totally freaked out. He was hysterical. He’s grabbing all this stuff.

His friend is like, “You’ve got to submit that.”

Lindsey Zuloaga: Yeah, they love that drama. We’ve seen some funny things. We have an internal hall of fame of some interviews we’ve seen with weird drunk uncles in the background. What we’ve also seen is some people have — I don’t know if maybe someone else set it up and they didn’t know or if it’s some default setting — but an anonymity mask appears on their computer. I don’t think they know, but in the interview they are a cat, a cartoon cat, and its mouth is moving with their mouths. I’m like, “What is this?” Those have been flagged because we can’t recognize this human face in this video.

Mike Delgado: That is so funny. Oh my gosh. That’s hilarious. Okay, Lindsey, we end our episodes with the same questions. So we go through these pretty quickly. The first one is, what is your favorite programming language?

Lindsey Zuloaga: Python.

Mike Delgado: We have a lot of people in our community who are studying data science. They want to get into the industry, but they need advice. I’m curious about what advice you’d give to somebody who is young and aspiring to be a data scientist.

Lindsey Zuloaga: There are so many resources online and so many great books, a lot of online courses. The one from Andrew Ng, I think at Stanford, is maybe on Coursera.

Mike Delgado: You need the right course, yeah.

Lindsey Zuloaga: That one I really liked, but there are a lot of resources. Also, like I said before, don’t underestimate the value of connections. I think most cities have data science meetups. It depends where you are, but look at meetup.com. Here in Salt Lake, we have many events focused on deep learning, machine learning, and data science. Just going in to events like that and talking to people is huge and then just getting experience. There’s a lot of cool stuff to do online, like cowbell competitions, where you can try to predict something, see how you did. There are tutorials to walk you through a lot of those types of problems. Like anything, it’s just practice, practice.

Mike Delgado: I think a key here for those who are listening and want to join data science is you need to prove yourself, be hungry, keep learning, get involved in different hacking competitions just so you can keep ahead, and keep showing your interest so when you do get that job interview, you have a whole bunch of stuff to show how much you love data science.
Last question. There are a lot of unknowns about AI in the future and a lot of scary tabloid headlines about rogue robots taking over the world. Curious about your view on how AI is going to impact us in the future.

Lindsey Zuloaga: I think it’s a really interesting topic, and I don’t know what’s going to happen exactly, so I don’t take a strong position. But my gut feeling is that like most technology we’ve been afraid of in the past, it turned out fine usually. There are uses that are good and evil for all technology. Like so many times in the past when people have been afraid of new technology, I think we’ll end up in a similar situation where there’s some bad things and there’s some scary things, but overall it’s going to be really great for humanity and make the Earth a better place to live. That’s my hunch but not to say that there are no fears. We need to be careful going forward.

Mike Delgado: The last question is, where can everyone learn more about you and your work?

Lindsey Zuloaga: I’m on Twitter @LindseyZuloaga. I don’t tweet that much, but I should more.

Mike Delgado: I was going to say I’ll make sure to put a link to your LinkedIn profile so people can follow you there and also to your Twitter account, and the URL for those listening to the podcast is just ex.pn/lindsey. That will bring you over to the Experian blog, where this video interview will be, along with the podcast episode and a full transcription for those who prefer just to read the interview. I’ll have links going to Lindsey’s profile so you can follow her.

Lindsey, thank you so much for being part of #DataTalk. It was a blast chatting with you, fascinating learning about the work you’re doing. It’s awesome to hear HireVue is going to be improving the whole hiring process, helping to eliminate bias.

Lindsey Zuloaga: Thanks so much for having me. It’s fun.

Mike Delgado: Awesome. Take care.

About Lindsey Zuloaga

Dr. Lindsey Zuloaga is the Director of Data Science at HireVue. She earned her Bachelor of Science degree in Applied Physics from the University of Utah and her Ph.D. in Applied Physics from Rice University.

Check out our upcoming data science live video chats.